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Abstract. To be able to provide better support for collaborative learning in In¬ 
telligent Tutoring Systems, it is important to understand how collaboration pat¬ 
terns change. Prior work has looked at the interdependencies between utteranc¬ 
es and the change of dialogue over time, but it has not addressed how dialogue 
changes during a lesson, an analysis that allows us to investigate the adaptivity 
of student strategies as students gain domain knowledge. We address this ques¬ 
tion by analyzing the shift in types of collaborative talk occurring within a sin¬ 
gle session and in particular how they relate to errors for 26 4 th and 5 th grade 
dyads working on a fractions tutor. We found that, over time, the frequency of 
interactive talk and errors both decrease in dyads working together on concep¬ 
tual problems. Although interactive talk is often held as a gold standard in col¬ 
laboration, as students become more proficient, it may not be as important. 
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1 Introduction 

As students work together on collaborative problem solving, their behaviors may 
change over time. To understand how to best support learning with the use of tech¬ 
nology, it is important to take this change into account so that the system can accu¬ 
rately adapt to the change in behaviors. This paper addresses changes in behavior that 
may be seen over a short period of time, namely, changes within a single collaborative 
learning session, which are often ignored by using an aggregate score for process data 
measures collected over an entire session. We focus our analysis on students’ dia¬ 
logue and how this is related to behavior in an Intelligent Tutoring System (ITS) de¬ 
signed to support collaborative learning. 

Past work with Computer Supported Collaborative Learning has focused on chang¬ 
es over time based on interdependencies within the dialogue [1], [4], [7] and how 
communication changes permanently over long periods of time [3], [6]. However, 
neither of these perspectives takes into account how behaviors and strategies may 
change temporarily within a single lesson as students learn, which may show how 



language is being used as a tool for learning rather than showing the learning of col¬ 
laboration strategies. For our study, we analyzed collaborative dialogue within a sin¬ 
gle lesson. Within an ITS, we often use a decrease in error rate as an indication of 
learning. As students make fewer errors, they may feel less of a need to discuss the 
answer and have less interactive talk. By using errors, we are able to investigate how 
students adapt their use of interactive talk as needed to help them through impasses. 
We analyzed how interactive talk changed between the first and second half of the 
session and how these changes were correlated to changes in error rates. 


2 Methods 

We analyzed data from a study in a school in which 26 4 th and 5 th grade dyads, paired 
within a grade (i.e., 13 4 th and 13 5 th grade dyads), engaged in a problem-solving ac¬ 
tivity geared towards either conceptual or procedural knowledge of fractions using a 
collaborative ITS [5], The problem sets offered standard ITS support (immediate 
feedback and hints) as well as embedded collaboration scripts supported through three 
different collaboration features: assigned roles, individual information, and cognitive 
group awareness. Each student had their own view of the collaborative ITS while they 
were synchronously working on a problem and communicated through Skype (audio 
only). Each dyad worked with the tutor for 45 minutes in a pull-out design. 

We coded the collaborative dialogue using a rating scheme with four major rating 
categories: interactive dialogue, constructive dialogue, constructive monologue, and 
other. For our analysis, we focused on the interactive dialogue category, in which 
students engage in actions such as co-construction and sequential construction, be¬ 
cause it aligns with ICAP’s joint dialogue pattern and is held as the gold standard for 
collaboration [2]. Our rating scheme was developed to look at groups of utterances 
associated with subgoals (i.e., a group of steps that all are for the same goal within a 
problem) to account for the interactions between students. An inter-rater reliability 
analysis was performed to determine consistency between two raters (Kappa= 0.72). 

In our analysis, we used two different error counts. The first was the total number 
of errors made on a subgoal by either student in the dyad. The second error count was 
the number of steps within the given subgoal that contained errors. For all variables 
the change was calculated by subtracting the first half value from the second half 
value. Since each dyad completed a different number of subgoals, from 25 to 143, the 
first and second halves of the session were calculated by dividing the total number of 
subgoals completed in half. All variables used in analysis were calculated in propor¬ 
tion to the number of subgoals that were completed. 


3 Results 

Because the procedural and conceptual tutor problems were fundamentally different 
with regard to the types of knowledge that were being learned, each of these condi¬ 
tions was treated separately. To test our hypothesis that the percent of subgoals where 
interactive talk occurs will change from the first to the second half of the session, we 



conducted two paired t-tests, one for the procedural condition and one for the concep¬ 
tual condition. For the procedural condition, there was no significant difference in the 
amount of interactive talk between the first (M= 0.24, SD = 0.18) and second half of 
the session (M = 0.21, SD = 0.19), t(12) = 1.23, p = 0.24. For the conceptual condi¬ 
tion, there was a significant decrease in the amount of interactive talk between the 
first (M= 0.32, SD = 0.20) and second half of the session (M= 0.19, SD = 0.12), i(12) 
= 3.50, p < 0.01. There was also a significant increase in the amount of other talk 
between the first and second half of the session, t(12) = -2.75, p < 0.05. 

To better understand how the change in interactive talk may be related to students’ 
problem-solving behaviors with the ITS, we analyzed how our two error measures 
differed between the first and second half of the session. We used a paired t-test to 
compare the two time points. In the procedural condition, there was no significant 
difference in the total number of errors made between the first (M = 1.27, SD = 0.63) 
and second half of the session (M = 1.44, SD = 1.02), t(12) = -0.87, p = 0.40, and 
there was no significant difference between the number of steps where errors occurred 
(M= 0.60, SD = 0.24, M= 0.53, SD = 0.26), t(12) = 1.48, = 0.17. In the conceptual 

condition, there was a significant decrease in the total number of errors that were 
made between the first (M = 3.46, SD = 2.84) and the second half (M = 1.69, SD = 
0.96), t( 12) = 2.36, p < 0.05 and there was also a significant decrease in the number of 
steps where an error occurred (M = 0.72, SD = 0.15; M = 0.58, SD = 0.17), 1(12) = 
2.45, p < 0.05. Thus, in the conceptual condition, we see a decrease from the first to 
the second half in both interactive talk and problem-solving errors. 

To better understand this relationship, we computed two Pearson’s correlations. 
Outliers were removed based on being more than 3 standard deviations away from the 
mean. First, we analyzed the relationship between the change in interactive talk and 
the change in the number of steps with errors. There was a strong positive correlation 
between the change in interactive talk and the change in the steps with errors, r(10) = 
0-59, p < 0.05. Thus, both interactive talk and the number of incorrect steps decreased 
over time and these decreases were strongly associated with each other. The second 
relationship that we analyzed was the correlation between the change in interactive 
talk and the change in total errors. There was a negative correlation between the 
change in interactive talk and the change in total errors with marginal statistical sig¬ 
nificance, r(10) = -0.53, p = 0.07. This result indicates that dyads with a greater de¬ 
crease of interactive talk from the first to second half of the session had less of a de¬ 
crease in the total errors made. 

4 Discussion and Conclusion 

Through our analysis we found that in the conceptual condition, students who had a 
greater decrease in the amount of interactive talk tended to have a smaller decrease in 
the number of errors that they made, but students who tended to have a greater de¬ 
crease in the amount of interactive talk tended to have a greater decrease in the num¬ 
ber of steps where errors occurred. These results indicate that interactive talk may not 
be necessary for students to correctly solve a problem, which is supported by the posi¬ 
tive correlation of decreased interactive talk and decreased steps with errors, but in- 



stead interactive talk may be a tool students can use when struggling. Interactive talk 
may not be so critical for correct problem-solving performance once students become 
more proficient, but it is helpful when students make errors. The negative correlation 
with the total errors would suggest other behaviors are occurring when students make 
multiple errors on the same step and where an area for future work. We did not find 
these same patterns for the procedural condition indicating that there may be different 
productive learning behaviors for procedural knowledge. The different results may 
indicate that for procedural knowledge there is less of a need for the deep understand¬ 
ing that is associated with interactive talk to overcome errors. 

To be able to provide better support for learning through collaborative ITSs, it is 
important to understand how the collaboration may change as students work together. 
Although interactive talk is often held as the gold standard in collaboration [2], our 
results suggest that interactive talk may not be necessary for the entire duration of a 
collaborative session, but only when errors occur. It would follow that a decrease in 
interactive talk over time is not necessarily an indication that the quality of the collab¬ 
oration is deteriorating. Rather, it may signal that the interactive talk is invoked in an 
adaptive manner, specifically to deal with impasses. By analyzing the change in col¬ 
laborative dialogue over the course of a lesson, we were able to clarify how interac¬ 
tive talk and in-tutor learning relate to each other. By analyzing shorter time intervals 
than halves for future work, we may be able to build upon our findings presented in 
this paper and discern smaller patterns that may be present in the process data. 
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